UFO Sightings - Data Analysis project¶
Prepared by Kourosh Khedri¶
1- Exploring Dataset¶
loading the dataset with pandas dataframe and checking how the first and last 10 rows look like:¶
In [1]:
import pandas as pd
In [2]:
ufo_data = pd.read_csv("nuforc_reports.csv")
In [7]:
ufo_data.head(10)
Out[7]:
In [8]:
ufo_data.tail(10)
Out[8]:
Checking if there are missing values exists in the dataset and what is the shape of data table:¶
In [10]:
ufo_data.isnull().sum()
Out[10]:
In [11]:
ufo_data.shape
Out[11]:
2- Data Cleaning¶
keeping the columns that I would like to clean and analyze:¶
In [12]:
# Leaving only the necessary columns
df1 = ufo_data[['city', 'state', 'date_time', 'shape', 'text']]
In [13]:
df1.head(5)
Out[13]:
Missing values treatment:¶
In [14]:
# Removing rows with missing values
df1 = df1.dropna(axis=0).reset_index(drop=True)
In [16]:
df1.shape
Out[16]:
Other data cleaning considerations:¶
In [17]:
# Fixing an abbreviation duplication issue
df1['state'] = df1['state'].apply(lambda x: 'QC' if x=='QB' else x)
In [18]:
# Creating a list of Canadian provinces
canada = ['ON', 'QC', 'AB', 'BC', 'NB', 'MB',
'NS', 'SK', 'NT', 'NL', 'YT', 'PE']
In [19]:
# Creating new columns: `country`, `year`, `month`, and `time`
df1['country'] = df1['state'].apply(\
lambda x: 'Canada' if x in canada else 'USA')
df1['year'] = df1['date_time'].apply(lambda x: x[:4]).astype(int)
df1['month'] = df1['date_time'].apply(lambda x: x[5:7]).astype(int)
df1['month'] = df1['month'].replace({1: 'Jan', 2: 'Feb', 3: 'Mar',
4: 'Apr', 5: 'May', 6: 'Jun',
7: 'Jul', 8: 'Aug', 9: 'Sep',
10: 'Oct', 11: 'Nov', 12: 'Dec'})
df1['time'] = df1['date_time'].apply(lambda x: x[-8:-6]).astype(int)
In [22]:
# Dropping an already used column
df1 = df1.drop(['date_time'], axis=1)
In [23]:
# Dropping duplicated rows
df1 = df1.drop_duplicates().reset_index(drop=True)
In [24]:
df1.head(20)
Out[24]:
In [25]:
round(df1['country'].value_counts(normalize=True)*100)
Out[25]:
96% of UFO sightings are related to USA¶
3- Data Visualization¶
Visualizing frequency of UFO occurences in each month:¶
In [26]:
import matplotlib.pyplot as plt
import seaborn as sns
In [27]:
# Creating a series object for UFO occurences by month, in %
months = df1['month'].value_counts(normalize=True)\
[['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']]*100
In [28]:
# Defining a function for creating and customizing a figure in matplotlib
def create_customized_fig():
fig, ax = plt.subplots(figsize=(12,6))
plt.title('UFO occurences by month, %', fontsize=27)
plt.ylim(0,15)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
ax.tick_params(bottom=False)
sns.despine()
return ' '
In [30]:
# PLOTTING
create_customized_fig()
# Creating a stem plot
plt.stem(months.index, months)
plt.show()
It can be observed the majority of UFO sightings in the USA and Canada are related to summer-autumn seasons, with a maximum around 12% in July, while in the winter-spring period there is much less activity, with a minimum 5% in February.¶
UFO shape frequency distribution, to check whether some shapes are more common than the others:¶
In [31]:
# Creating a series of shapes and their frequencies
# in ascending order
shapes = df1['shape'].value_counts(normalize=True,
ascending=True)*100
fig, ax = plt.subplots(figsize=(12,9))
# Creating a vertical stem plot
plt.hlines(y=shapes.index,
xmin=0, xmax=shapes,
color='slateblue',
linestyle='dotted', linewidth=5)
plt.plot(shapes, shapes.index,
'*', ms=17,
c='darkorange')
plt.title('UFO shapes by sighting frequency, %', fontsize=29)
plt.xlim(0,25)
plt.yticks(fontsize=20)
plt.xticks(fontsize=20)
ax.tick_params()
sns.despine()
plt.show()
According to the witnesses, UFO can take a wide range of incredible forms, including diamonds, cigars, chevrons, teardrops, and crosses. The far most frequent form (22%), however, is described as just a light.¶
Visualizing most common used words in USA and Canada via Word Cloud:¶
In [32]:
from wordcloud import WordCloud, STOPWORDS
# Gathering sighting descriptions from all American witnesses
text = ''
for t in df1[df1['country']=='USA'].loc[:, 'text']:
text += ' ' + t
fig = plt.subplots(figsize=(10,10))
# Creating a basic word cloud
wordcloud = WordCloud(width=1000, height=1000,
collocations=False).generate(text)
plt.title('USA collective description of UFO', fontsize=27)
plt.imshow(wordcloud)
plt.axis('off')
plt.show()
# Saving the word cloud
wordcloud.to_file('wordcloud_usa.png')
Out[32]:
The most common words are "light", "object", and "sky", followed by "bright", "time", "moving", "white", "red", "craft", "star". Among the most frequent words, there are some low-informative ones, like "one", "second", "saw", "see", "seen", "looked", etc. We can assume that American witnesses mostly observed bright craft objects of white or red color, moving in the sky and emitting light.¶
In [33]:
# Gathering sighting descriptions from all Canadian witnesses
text = ''
for t in df1[df1['country']=='Canada'].loc[:, 'text']:
text += ' ' + t
# Creating a user stopword list
stopwords = ['one', 'two', 'first', 'second', 'saw', 'see', 'seen',
'looked', 'looking', 'look', 'went', 'minute', 'back',
'noticed', 'north', 'south', 'east', 'west', 'nuforc',
'appeared', 'shape', 'side', 'witness', 'sighting',
'going', 'note', 'around', 'direction', 'approximately',
'still', 'away', 'across', 'seemed', 'time']
fig = plt.subplots(figsize=(10,10))
# Creating and customizing a word cloud
wordcloud = WordCloud(width=1000, height=1000,
collocations=False,
colormap='cool',
background_color='yellow',
stopwords=STOPWORDS.update(stopwords),
prefer_horizontal=0.85,
random_state=100,
max_words=100,
min_word_length=3).generate(text)
plt.title('Canadian collective description of UFO', fontsize=27)
plt.imshow(wordcloud)
plt.axis('off')
plt.show()
# Saving the word cloud
wordcloud.to_file('wordcloud_canada.png')
Out[33]:
It seems that the descriptions given by Canadian people are rather similar to those from Americans, with the addition of some other frequent words: "orange", "plane", "night", "minutes", "seconds", "cloud", "flying", "speed", "sound". We can assume that Canadians witnessed bright craft objects, of white, red, or orange color, mostly at night time, moving/flying in the sky and emitting light and, probably, sound. At first, the objects looked like stars, planes, or clouds, and the whole process lasted several seconds to minutes.¶
The difference between Canadian and American collective descriptions can be partially explained by adding some more words to the stopword list. Or, maybe, “Canadian” aliens are really more orange, plane- or cloud-like, and noisy.¶
Visualizing UFO sighting frequencies by each state in USA:¶
In [35]:
pip install squarify
In [36]:
import squarify
# Extract the data
states = df1[df1['country']=='USA'].loc[:, 'state'].value_counts()
fig = plt.subplots(figsize=(12,6))
# Creating a treemap
squarify.plot(sizes=states.values, label=states.index)
plt.title('UFO sighting frequencies by state, the USA', fontsize=27)
plt.axis('off')
plt.show()
Looks like California is a real extraterrestrial base in the USA! It’s followed with a big gap by Florida, Washington, and Texas, while the territories of District of Columbia and Puerto Rico are visited by UFO very rarely.¶
Visualizing what time the most/least aliens were seen:¶
In [37]:
import matplotlib
# Extracting the data
hours = df1['time'].value_counts()
# Creating a list of colors from 2 matplotlib colormaps
# `Set3` and `tab20`
cmap1 = matplotlib.cm.Set3
cmap2 = matplotlib.cm.tab20
colors = []
for i in range(len(hours.index)):
colors.append(cmap1(i))
if cmap2(i) not in colors:
colors.append(cmap2(i))
fig = plt.subplots(figsize=(12,6))
# Creating and customizing a treemap
squarify.plot(sizes=hours.values, label=hours.index,
color=colors, alpha=0.8,
pad=True,
text_kwargs={'color': 'indigo',
'fontsize': 20,
'fontweight': 'bold'})
plt.title('UFO sighting frequencies by hour', fontsize=27)
plt.axis('off')
plt.show()
The respondents from our dataset mostly observed UFO in the time range from 20:00 till 23:00, or, more generally, from 19:00 till midnight. The least “UFO-prone” hours are 07:00–09:00. However, it doesn’t necessarily mean the “lack of aliens” in certain hours of the day and instead can be explained more pragmatically: usually people have free time in the evening after work, while in the morning the majority of people are going to work and are a bit too immersed in their thoughts to notice interesting phenomena around them.¶
Visualizing UFO shape appearances between 2015 - 2019:¶
In [39]:
pip install matplotlib-venn
In [40]:
from matplotlib_venn import *
# Creating the subsets for crosses and cigars
crosses = df1[(df1['shape']=='cross')&\
(df1['year']>=2015)&(df1['year']<=2019)].loc[:, 'city']
cigars = df1[(df1['shape']=='cigar')&\
(df1['year']>=2015)&(df1['year']<=2019)].loc[:, 'city']
fig = plt.subplots(figsize=(12,8))
# Creating a Venn diagram
venn2(subsets=[set(crosses), set(cigars)],
set_labels=['Crosses', 'Cigars'])
plt.title('Crosses and cigars by number of cities, 2015-2019',
fontsize=27)
plt.show()
In the period from 2015 till 2019 inclusive, there were 18 cities in North America where both crosses and cigars were registered. In 79 cities, only crosses were observed (from these 2 shapes), in 469 – only cigars.¶
In [41]:
# Creating a subset for diamonds
diamonds = df1[(df1['shape']=='diamond')&\
(df1['year']>=2015)&(df1['year']<=2019)].loc[:, 'city']
# Creating a list of subsets
subsets=[set(crosses), set(cigars), set(diamonds)]
fig = plt.subplots(figsize=(15,10))
# Creating a Venn diagram for the 3 subsets
venn3(subsets=subsets,
set_labels=['Crosses', 'Cigars', 'Diamonds'],
set_colors=['magenta', 'dodgerblue', 'gold'],
alpha=0.3)
# Customizing the circumferences of the circles
venn3_circles(subsets=subsets,
color='darkviolet', alpha=0.9,
ls='dotted', lw=4)
plt.title('Crosses, cigars, and diamonds \nby number of cities, 2015-2019', fontsize=26)
plt.show()
Hence, in the period of interest there were 6 cities in North America where all 3 shapes were registered, 66 cities – where only cigars and diamonds, 260 – only diamonds, etc.¶
This is to check what are those 6 cities that have in common for all the 3 shapes:¶
In [42]:
print(set(crosses) & set(cigars) & set(diamonds))
All of them are located in the USA.¶
Box Plot Visualization¶
In [43]:
df1[df1['country']=='Canada'].loc[:, 'state'].value_counts()[:3]
Out[43]:
In [59]:
# Extracting the data for cylinders and cones
# from California and Ontario
CA_ON_cyl_con = df1[((df1['state']=='CA')|(df1['state']=='ON'))&((df1['shape']=='cylinder')|(df1['shape']=='cone'))]
fig = plt.subplots(figsize=(12,7))
sns.set(style='white')
# Creating swarm plots
sns.swarmplot(data=CA_ON_cyl_con, x='year', y='state', palette=['deeppink', 'blue'])
# Creating box plots
sns.boxplot(data=CA_ON_cyl_con, x='year', y='state', palette=['palegreen', 'lemonchiffon'])
plt.title('Cylinders and cones in California and Ontario', fontsize=29)
plt.xlabel('Years', fontsize=18)
plt.ylabel('States', fontsize=18)
sns.despine()
plt.show()
The following observations can be made:¶
- Since the numeric variable in question (year) is an integer, the data points are aligned.¶
- Both subsets are quite different in terms of their sample size.¶
- The Californian subset is heavily left-skewed and contains a lot of outliers.¶
- None of the box plots gives us an idea about the underlying data distributions. In the case of the Californian subset, there are a lot of conic or cylindric UFO related to the 3rd quartile of the distribution, as well as to the most recent year, 2019.¶
- We definitely should add to our “wish list” the possibility to distinguish between cylinders and cones for each subset.¶
This is to exclude the outliers from the visualization:¶
In [60]:
fig = plt.subplots(figsize=(12,7))
# Creating swarm plots
sns.swarmplot(data=CA_ON_cyl_con, x='year', y='state', palette=['deeppink', 'blue'], hue='shape')
# Creating box plots
sns.boxplot(data=CA_ON_cyl_con, x='year', y='state', palette=['palegreen', 'lemonchiffon'])
plt.title('Cylinders and cones in California and Ontario', fontsize=29)
plt.xlim(1997,2020)
plt.xlabel('Years', fontsize=18)
plt.ylabel('States', fontsize=18)
plt.legend(loc='upper left', frameon=False, fontsize=15)
sns.despine()
plt.show()
Both plots show that the predominant majority of UFO for these 2 subsets are cylinders. For the Californian subset, we can distinguish the years of particularly frequent occurences of cylindric/conic UFO: 2008, 2015, and 2019. Moreover, in 2015, we observe an unexpected boom of cones, despite they are much rarer in general.¶
Visualizing the UFO occurrences on the map:¶
In [62]:
pip install folium
In [63]:
import folium
from folium.plugins import HeatMap
In [64]:
df_plot = ufo_data[~ufo_data.city_latitude.isna()]
df_plot.shape
Out[64]:
In [65]:
# heatmap of locations for first overview
zoom_factor = 2 # inital map size
my_map_1 = folium.Map(location=[0,0], zoom_start=zoom_factor)
HeatMap(data=df_plot[['city_latitude', 'city_longitude']], radius=10).add_to(my_map_1)
my_map_1 # display
Out[65]:
we can see the location of each occurrence on the map and check where they were mostly focused.¶
Visualizing the trend between 1970 to 2019:¶
In [70]:
pd.pivot_table(df1,index='year',values='state',aggfunc='count').plot(figsize=(10,6))
plt.title('1970 - 2019')
Out[70]:
